[skills] chore: rename overlap and recompute skills with nemo mbridge prefix - #4031
Conversation
Signed-off-by: Chen Cui <chcui@nvidia.com>
|
/nvskills-ci |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| 1. **Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` first** — most | ||
| borderline OOMs are caused by memory fragmentation, not capacity. This | ||
| fixes it at zero cost. See @skills/perf-memory-tuning/SKILL.md. | ||
| fixes it at zero cost. See @skills/mbridge-perf-memory-tuning/SKILL.md. |
There was a problem hiding this comment.
Bug: This references @skills/mbridge-perf-memory-tuning/SKILL.md, but the actual directory is still skills/perf-memory-tuning/. This name was not renamed in this PR (and mbridge-perf-memory-tuning doesn't match the nemo-mbridge-* prefix pattern used elsewhere in this PR either). This is a broken cross-reference.
| fixes it at zero cost. See @skills/mbridge-perf-memory-tuning/SKILL.md. | |
| fixes it at zero cost. See @skills/perf-memory-tuning/SKILL.md. |
| - Combining `mlp` + `core_attn` is slightly worse than `mlp` alone | ||
| - For this workload, the actual OOM fix was `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` | ||
| (memory fragmentation, not capacity). See @skills/perf-memory-tuning/SKILL.md. | ||
| (memory fragmentation, not capacity). See @skills/mbridge-perf-memory-tuning/SKILL.md. |
There was a problem hiding this comment.
Same broken reference — mbridge-perf-memory-tuning doesn't exist.
| (memory fragmentation, not capacity). See @skills/mbridge-perf-memory-tuning/SKILL.md. | |
| (memory fragmentation, not capacity). See @skills/perf-memory-tuning/SKILL.md. |
| if_still_oom: "add layernorm (cheap) or mlp (expensive but saves ~3 GB)" | ||
| last_resort: "recompute_granularity=full, recompute_method=uniform" | ||
| alternative: "see skills/perf-memory-tuning/ for VPP tuning and other memory strategies" | ||
| alternative: "see skills/mbridge-perf-memory-tuning/ for VPP tuning and other memory strategies" |
There was a problem hiding this comment.
Same issue — skills/mbridge-perf-memory-tuning/ doesn't exist. Should remain skills/perf-memory-tuning/ until that skill is actually renamed.
| alternative: "see skills/mbridge-perf-memory-tuning/ for VPP tuning and other memory strategies" | |
| alternative: "see skills/perf-memory-tuning/ for VPP tuning and other memory strategies" |
|
Review: Broken cross-references to non-existent skill directory -- Three places in the renamed nemo-mbridge-perf-activation-recompute skill now reference skills/mbridge-perf-memory-tuning/SKILL.md (SKILL.md lines 32, 132; card.yaml line 95). The actual directory is still skills/perf-memory-tuning/ -- it was not renamed in this PR, and the target name mbridge-perf-memory-tuning does not match the nemo-mbridge-* prefix convention used for the other renames. These are broken cross-references. See inline comments for suggested fixes. -- Stale references in files not touched by this PR: Per CONTRIBUTING.md, renames must update all references. Files still using old names: skills/perf-memory-tuning/SKILL.md (3 refs to perf-activation-recompute), skills/perf-cuda-graphs/SKILL.md and card.yaml (2 refs to perf-activation-recompute), docs/training/communication-overlap.md (3 refs to perf-expert-parallel-overlap, 2 refs to perf-moe-comm-overlap), docs/training/activation-recomputation.md (1 ref to perf-activation-recompute), docs/training/moe-optimization.md (1 ref to perf-moe-comm-overlap). These should be updated. -- Suggested test cases: No perf tests impacted. |
yaoyu-33
added
area:misc
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
|
/nvskills-ci |
Signed-off-by: Chen Cui <chcui@nvidia.com>
|
Scoped this signing PR back down to the renamed skill directories only. The cross-skill reference updates that touched old skill directories were moved to draft cleanup PR #4044 so NVSkills should not validate unrelated old skill dirs in this batch. |
|
/nvskills-ci |
Signed-off-by: Chen Cui <chcui@nvidia.com>
|
Kept the activation-recompute references pointed at the final nemo-mbridge memory-tuning skill name so this batch does not pull the old memory-tuning skill into validation. Rerunning NVSkills now. |
|
/nvskills-ci |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
|
/ok to test 1921fa5 |
Signed-off-by: Chen Cui <chcui@nvidia.com>
… prefix (NVIDIA-NeMo#4031) Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com> Co-authored-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com> Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>
3 participants
Summary
nemo-mbridge-*prefix:nemo-mbridge-perf-expert-parallel-overlapnemo-mbridge-perf-moe-comm-overlapnemo-mbridge-perf-activation-recomputelicense: Apache-2.0frontmatter for these public skills.evals/evals.jsonfiles where needed for NVSkills signing.docs/skills-index.mdreferences for the renamed skills.Validation
SKILL.mdfrontmatter names.license: Apache-2.0.evals/evals.jsonfiles.SKILL.mdfiles are <= 500 lines.git diff --checkpassed.uv run --no-sync pre-commit run --all-filespassed.